Hierarchical AI enables global interpretation of culture plates in the era of digital microbiology

Full Laboratory Automation is revolutionizing work habits in an increasing number of clinical microbiology facilities worldwide, generating huge streams of digital images for interpretation. Contextually, deep learning architectures are leading to paradigm shifts in the way computers can assist with difficult visual interpretation tasks in several domains. At the crossroads of these epochal trends, we present a system able to tackle a core task in clinical microbiology, namely the global interpretation of diagnostic bacterial culture plates, including presumptive pathogen identification. This is achieved by decomposing the problem into a hierarchy of complex subtasks and addressing them with a multi-network architecture we call DeepColony. Working on a large stream of clinical data and a complete set of 32 pathogens, the proposed system is capable of effectively assist plate interpretation with a surprising degree of accuracy in the widespread and demanding framework of Urinary Tract Infections. Moreover, thanks to the rich species-related generated information, DeepColony can be used for developing trustworthy clinical decision support services in laboratory automation ecosystems from local to global scale.

In this manuscript, the authors developed a hierarchical multi-network capable of handling identification, quantitation and interpretation stages, from the single colony to the whole plate level, in the challenging context of urinary tract infections.
Working on a large stream of clinical data and a complete set of 32 pathogens, the proposed system proved to be capable of effectively assisting plate interpretation with a surprising degree of accuracy.Thanks to the rich species-related generated information, the authors claim that DeepColony can be used for developing trustworthy clinical decision support services in lab automation ecosystems from the local to the global scale.
Although I do not have specific knowledge on hierarchical AI, deep learning and Convolutional Neural Networks, I read with interest this manuscript from the view point of a clinical microbiologist.
As a general view, the observation that the overall accuracy was high, is very promising.
I have some general comments for the authors consideration: -Only blood agar plates were used, on which almost all urinary bacteria are grown and the discrimination of colonies is rather difficult.Why the selective Mac Conkey plates were not used at all, particularly when UTIs were investigated, in which mostly Gram-negatives are yielded and the majority of Gram-positive colonies are contaminants?In Mac Conkey plates, other properties, such as lactose fermentation would help the identification.
-The quality of plating in figure 5 is quite low; the growth in two of the plates is rather confluent, the of pure colonies would be needed for such a system to perform appropriately.Development of such strategy would be important for the system in the future.
-I would like to see 1-2 figures of real-life plates from positive urinary cultures (with one or two different and not so many different colony types), so that the reader may estimate the accuracy of colonies' enumeration and reporting.
-The paragraph lines 294-316 are mainly general, introductory comments and not methods.
-It would be of considerable interest if other types of clinical samples, further than UTI ones, were also tested.
Reviewer #2 (Remarks to the Author): I want to start this review with a clear statement that I have a pure machine learning background with a few papers in automatic microbiology.Moreover, I assume that Nature Communication is not a purely microbiological journal, like Trends in Microbiology, and articles presented in this journal should be directed to a wider audience than only microbiological experts.Therefore, my review should be considered as a review from a potential Nature Communication reader with a machine learning background.
Overall, it is clear to me that the authors have extensive knowledge in microbiology and have published many papers on this topic.However, the article is very unclear for somebody who does not have such an experience.The descriptions are too long and very complicated, and I guess only a few people in the world who are on the same level of expertise as the authors would be able to read this article and get something interesting from it.
Taking this into consideration, I decided to read Intro, Results, and Discussion very fastly and concentrate on the Method section, which I thought should be of interest to somebody like me -i.e. machine learning expert who cooperate with microbiologists.
Unfortunately, the Method section was also unclear to me. Figure 1a (Overview of the DeepColony system) brings more confusion than clarifications.There is not a single figure of network architecture.Reading the paragraph "Single colony identification", I first thought that the authors use simple CNN architecture, but then "Context-based Identification" appeared with "Siamese" networks, and the description is too fuzzy to understand how they are used and why?The authors bring too many implementation details, like PreReLU, momentum, etc, which should be moved to supplementary materials.
In my opinion, the current version of the article should not be accepted, and the editor should ask the authors to rewrite the whole method section using best practices from other articles already published in Nature Communication, like this one: https://www.nature.com/articles/s41467-019-12898-9 This already-published article, in contrast to the reviewed one: -Has a shorter Introduction with a clear image presenting what is done in the article.
-Has a very clear Method section, which is understandable for machine learning experts.
If the article is rewritten, I will gladly analyze it again and give more comments on the method per se.Due to the current unclearness of the article, I, unfortunately, cannot do it.In this paper, a new system (namely global interpretation of diagnostic bacterial culture plates, including presumptive pathogen identification.) is proposed to deal with the problem of bacterial culture plate analysis within a digital microbiology research background.First, a dataset is built up, where 1351 plate images, 25213 colony images and 32 microorganism species are included.Then, a two-step method framework is introduced.In the first step, single colony identificatioon (level 0-2) is carried out.In the second step, a context-based indetification is done.Finally, different evaluation values are calculated to evaluate the performance of this system.My detailed comments are as follows: 1. Application value: As mentioned in the paper, "Full Laboratory Automation is revolutionizing working habits in a steadily increasing number of clinical microbiology facilities worldwide, generating huge flows of digital images to interpret.",the application goal of this research is meaningful and would be useful for microbiology and medical data analysis.
2. Technique value: From my private feeling, I like the idea of this system very much.However, from an objective point of technology, the methodological contribution of this paper is really limited, where only some existing machine learning methods are simply modified and used to solve this special digital microbiology problem.My suggestions are below: (1) In "Single colony identification (level 0 -2)": 1) The proposed CNN can achieve the purpose of colony identification, however, the architecture is kind of out-of-date.
2) In Fig. 3(a), the proposed model seems like a traditional VGG-based model, which is lack of novelty.
3) Besides, the normalization (such as BN) is missed in this model, and the reason should be given.

4)
The feature extraction capability of such a shallow neural network may be doubted, and the improvement of this model may improve the identification performance.5) Maxpooling can be applied for down-sampling, but may lost many meaningful features, try to find out another down-sampling method if possible.
(2) In "Context-based Identification (level 3)": 1) Similarly, the Siamese CNN is also an old approach for similarity comparison of two patches.
2) And there is no improvement for architecture and loss function design of the proposed model.
3) Besides, an existing mean shift clustering is applied here, which shows few novelty value for the proposed method.
3.Data quality: In this paper, a large labeled clinical dataset is created for the training of DeepColony at the colony level.Starting from 1351 unique plate images, a dataset of 26213 isolated colony images was produced.These represent 32 UTI bacterial and fungal species, constituting 98% of the species that have been observed in three months of the clinical routine of a large CML, and are represented in their clinical variability.So, the quality of this dataset is very good.4. Experimental result: the experimental results are good.However, there are not enough contrast experiments to show the special advantages of the proposed method.For example, the transformerbased models can be applied to compared with the proposed CNN-based models; for level 3, several traditional similarity evaluation indices such as SSIM should be applied as a contrast experiment.

Figure quality:
The figures in this paper are not vector images an not clear to show complex contents, such as Fig. 2, 3, 5.

Table quality: OK.
7. Equation and mathematic quality: Some mathematical symbols are not professional.For example, in "Single colony identification", times should be "×" but not "x".
8. Language quality: The English presentation in this paper is good.9. Reference quality: There many existing work about microorganism image analysis, but the authors did not read.I recommend the authors to read but not have to cite: [1] Microorganism image biovolume measurement: A Comprehensive Survey with Quantitative Comparison of Image Analysis Methods for Microorganism Biovolume Measurements; [2] Microorganism image analysis using deep learning: Applications of artificial neural networks in microorganism image analysis: a comprehensive review from conventional multilayer perceptron to popular convolutional neural network and potential visual transformer.10.Overall evaluation: I think the application motivation and writing quality of this review paper is good, but the tech novelty and methodological contributions of the paper are very limited.

REVIEWER COMMENTS
Reviewer #1 (Remarks to the Author): My comments were addressed adequately.
Although the manuscript is a bit complicated and not very applicable in the clinical practice, the idea is valid as a starting point and worths publication.
Reviewer #2 (Remarks to the Author): Firstly, I want to thank the authors for their thoughtful responses to my review.I think they put a lot of work into preparing the new version of the manuscript.Unfortunately, the current version is still far from perfect due to the following reasons: -Paragraph "DeepColony CNN architectures and training" should contain information about the overall pipeline with references to detailed descriptions in the following paragraphs.In fact, this paragraph should reference some pseudo-code that describes the pipeline more formally.Now, it is still unclear what are the successive steps and how they depend on each other... -Paragraphs after "DeepColony CNN architectures and training" still contain too many details about hyperparameters etc.It should be moved to the supplementary materials.This way, paragraphs like "Single colony identification" should limit to one sentence about using CNN architecture to identify a single colony.
-More effort should be put into describing the pipeline because, as I understand, it is the main achievement of the paper.Using CNN is no novelty whatsoever.
-Considering the paragraph "Nonlinear similarity-driven embedding", I still have no idea why the authors use Siamese networks.Maybe it will clarify by pseudo-code.
-When it comes to Fig. 1 (the most important figure of the paper), I really enjoy parts a, f, and g.However, parts b-e are still unclear.They should be moved to separate figures and redrawn.
-First three paragraphs from section Methods should be in the other section called, e.g., Dataset.
Reviewer #3 (Remarks to the Author): After checking the updated paper, I found that all my comments were solved and the quality of this paper had been improved a lot to close to a final publication.

REVIEWERS' COMMENTS
Reviewer #2 (Remarks to the Author): Reviewer #3 (Remarks to the Author): Journal: nature communications Title: Hierarchical AI enables global interpretation of culture plates in the era of digital microbiology Number: NCOMMS-22-46898-T

Figure 1
Figure1looks really clear and impressive.I think it will be great figure to show in any presentations that the authors will give about the paper.And, I am glad I could help with improving it.Paragraph DeepColony architecture also improved a lot.I like the overall description at the beginning and details limited to the most important things.Pseudo code also looks fine.